Skip to content

feat(blaze): prune checkpoint history - #2474

Open
WeissonHan wants to merge 1 commit into
alibaba:mainfrom
WeissonHan:feature/blaze/checkpoint-prune-draft-v1
Open

feat(blaze): prune checkpoint history#2474
WeissonHan wants to merge 1 commit into
alibaba:mainfrom
WeissonHan:feature/blaze/checkpoint-prune-draft-v1

Conversation

@WeissonHan

@WeissonHan WeissonHan commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Why

Blaze can commit and list checkpoints but has no way to reclaim the disk
space held by checkpoints that neither the current HEAD nor any explicitly
protected identifier can still reach. Operators need a bounded, restart-safe
way to remove unreachable checkpoint history without touching the active
capture chain.

What changed

  • Add POST /v1/sandboxes/{id}/checkpoints/prune. The request body is
    optional; when present it must be a JSON object with only a protected
    array of canonical ckpt-<uuid> identifiers. The response reports the
    removed count and identifiers.
  • Prune protects the current HEAD together with its full parent chain and the
    ancestor chains of every explicitly protected identifier. Unknown protected
    identifiers are rejected with HTTP 400 before any deletion; the endpoint
    otherwise returns HTTP 200 with the removed list (which may be empty).
  • Publish a named tombstone directory before renaming a candidate. The paired
    owner file records the sandbox, checkpoint, nonce, and directory identity
    so restart cleanup can identify each pruned candidate and finish removing
    it. renameat NOREPLACE prevents silently overwriting a concurrent
    candidate; identity revalidation between planning and rename rejects a
    catalog that has changed since the plan was built.
  • Extend classify_scratch_name to recognize .prune.<checkpoint>.<nonce>
    tombstone directories and their .owner files. remove_sandbox and the
    existing scratch cleanup already act on these categories, so restart
    cleanup or destroy removes any interrupted prune leftovers.
  • Serialize with capture and list through the existing per-sandbox operation
    lock and run the actual scan on the blocking pool.
  • Document the route, protection rules, response shape, tombstone flow, and
    restart behavior in the English and Chinese README and runtime user guide.

No /v1/instances compatibility route is added. No dependency on #2473,
#2475, or #2471.

Related issue

Follow-up on #2472 (capture and list). Delivers the pruning entry point that
was intentionally deferred.

User / Agent impact

Operators and agents can reclaim disk space held by unreachable checkpoint
branches without stopping or destroying the sandbox. The endpoint acquires
the same per-sandbox operation lock as capture and list, so callers do not
observe partial history changes.

Risk and compatibility

  • Public CLI, API, configuration, or documented behavior changed
  • Privileged or security-sensitive behavior changed
  • Cross-component contract changed
  • Migration or rollback guidance is needed

This adds one HTTP endpoint and two new scratch-name categories inside the
sandbox checkpoint namespace. Existing catalogs remain compatible: no new
files appear until an operator calls prune, and any interrupted prune leaves
only well-formed tombstone and owner files that the existing catalog
cleanup already handles.

Prune never touches the HEAD-reachable chain, so a caller with the wrong
protected list will at worst leave more history than intended; a repeat
call with an updated list is safe. The endpoint fails closed before deletion
on any protected identifier that is not currently committed.

Validation

Exact candidate:

An uninterrupted Linux x86_64 run on Rust 1.88.0 completed for the tree above.
It used a fresh source directory, a dedicated Cargo home, and an initially
empty task-exclusive target directory for each stage. Every Cargo stage ran
with --locked --offline.

Stage Result
cargo fmt --all -- --check pass
cargo metadata, default and all features pass
cargo build --workspace --all-targets, default and all features pass
cargo clippy --workspace --all-targets -- -D warnings, default and all features pass, no warnings
cargo test --workspace -- --test-threads=1 360 passed, 0 failed, 0 ignored
cargo test --workspace --all-features -- --test-threads=1 413 passed, 0 failed, 0 ignored
RUSTDOCFLAGS=-D warnings cargo doc --workspace --no-deps, default and all features pass, no warnings
scripts/docs-lint.sh pass
scripts/docs-link-check.py pass
git diff --check pass
source tree hash before and after the run unchanged

Focused prune coverage:

  • checkpoint_store::tests::prune_preserves_head_and_explicit_lineages proves
    the HEAD chain and every ancestor of a protected identifier survive while an
    unreachable branch is removed.
  • checkpoint_store::tests::prune_rejects_a_protected_id_that_is_not_committed
    proves an unknown protected identifier is rejected before deletion.
  • checkpoint_store::tests::prune_leaves_a_missing_sandbox_directory_untouched
    proves a sandbox without any checkpoint namespace remains untouched.
  • checkpoint_store::tests::prune_removes_owner_and_tombstone_after_completion
    proves the sandbox directory contains no prune scratch after a successful
    prune.
  • api::tests::prune_route_reports_no_removals_when_history_is_all_reachable
    proves the HTTP contract for a fully-reachable history.
  • api::tests::prune_route_rejects_unknown_body_fields proves the request
    body accepts only protected.
  • api::tests::prune_route_rejects_unknown_protected_checkpoint_as_bad_request
    proves a canonical but uncommitted protected identifier answers HTTP 400.
  • checkpoint_store::tests::prune_reports_partial_cleanup_when_post_tombstone_work_fails
    proves a failure after the tombstone rename returns PruneOutcome::PartialCleanup
    with the confirmed removed prefix, retains the prune scratch for later
    cleanup, and leaves the HEAD chain intact.
  • api::tests::prune_route_rejects_a_recovery_required_sandbox proves a
    recovery-required sandbox answers HTTP 409 instead of silently accepting a
    prune retry that would bypass retained scratch.

Documentation and rollback

The English and Chinese Blaze README and runtime user guide document the new
endpoint, protection rules, response shape, tombstone flow, and restart
behavior. No changelog entry is added because Blaze has not been released.

To roll back, stop the Blaze daemon and confirm that no sandbox has active
prune scratch (files starting with .prune. inside a sandbox checkpoint
directory). Revert this commit together with its API, checkpoint-store, and
documentation changes. Any tombstone or owner files left on disk are safe to
delete by hand once the daemon is stopped; the file provider will not
reference them after revert.

@github-actions github-actions Bot added component:blaze src/blaze scope:documentation ./docs/|./*.md|./NOTICE labels Aug 13, 2026
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 5804993 to 88921b2 Compare August 17, 2026 16:03
@WeissonHan
WeissonHan marked this pull request as ready for review August 17, 2026 16:07
@WeissonHan
WeissonHan requested a review from casparant as a code owner August 17, 2026 16:07
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Checkpoint history pruning: adds POST /v1/sandboxes/{id}/checkpoints/prune that protects the current HEAD chain and any explicitly listed lineages, publishes named tombstones before renaming candidates, and lets restart cleanup finish an interrupted prune. Rebased on merged #2472.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 88921b2483

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 88921b2 to 91a138e Compare August 17, 2026 16:21
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Prune with tightened BadRequest handling for uncommitted protected identifiers.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91a138e20b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs Outdated
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 91a138e to 06bd020 Compare August 18, 2026 01:08
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Prune now reports post-tombstone cleanup failures as PruneOutcome::PartialCleanup and the manager marks the sandbox RecoveryRequired.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06bd020fdc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
Comment thread src/blaze/crates/blazed/src/sandbox/checkpoint.rs
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 06bd020 to 9af5bdd Compare August 18, 2026 01:53
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex Review

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review Prune now classifies every post-rename uncertainty as PruneOutcome::PartialCleanup and rejects prune on a RecoveryRequired sandbox with HTTP 409.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9af5bdd952

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
Comment thread src/blaze/crates/blazed/src/checkpoint_store.rs Outdated
Add POST /v1/sandboxes/{id}/checkpoints/prune. The endpoint verifies the
committed catalog, protects HEAD together with its full parent chain and
the ancestor chains of any explicitly listed identifiers, and removes only
unreachable entries.

Publish a named tombstone before renaming a candidate directory so a
restart can identify the pruned candidate and finish removing it. The
paired owner file records the sandbox, checkpoint, nonce, and directory
identity, and restart cleanup already treats prune tombstones and owner
records the same way as capture staging leftovers.

Document the route, protection rules, and restart behavior in the English
and Chinese operator README and runtime user guide.

Signed-off-by: Weisson <Weisson@linux.alibaba.com>
@WeissonHan
WeissonHan force-pushed the feature/blaze/checkpoint-prune-draft-v1 branch from 9af5bdd to 9fcfb42 Compare August 18, 2026 02:26
@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review sweep_candidate now classifies every per-candidate outcome (Removed / Retained / RemovedWithRetainedScratch) so a partial success is never reported as a plain HTTP 500, and owner-publication failures clean up scratch or force RecoveryRequired.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@WeissonHan

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9fcfb424fb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

// Publishing the owner file creates prune scratch. Any failure between
// creation and the rename must remove that scratch, otherwise a later
// `Complete` result would contradict its documented invariant.
let owner = retain!(write_json_new(sandbox, &owner_name, &owner_record));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recover owner files when publication fails

When writing or syncing the owner fails after its O_EXCL creation—for example, due to ENOSPC or an I/O error—write_json_new returns an error here and retain! classifies it as Retained, so no cleanup path receives the file handle. With no earlier removals, the manager consequently returns HTTP 500 and leaves the sandbox Running while a partial .prune.*.owner file remains and later prune calls ignore it. Fresh evidence after the prior fix is that write_json_new still performs the fallible writes and sync_all internally at lines 1828–1833 before returning the owner handle; expose the created file for cleanup or classify this as recovery-required.

AGENTS.md reference: src/blaze/AGENTS.md:L32-L36

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:blaze src/blaze scope:documentation ./docs/|./*.md|./NOTICE

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant